Date: Sat, 28 Apr 90 04:50:43 +0200 From: klemets@sics.se Subject: NNTP for NOS To: tcp-group@ucsd.edu Because of this recent discussion about News support in NOS, I thought it would be a nice idea to implement NNTP in NOS. I have only done a client yet however. The client is controlled by the following commands: nntp addserver where is the hostname of a server from which you want to fetch the news. is the interval between polls, measured in seconds. nntp dropserver nntp kick nntp listserver What these commands do should be obvious. nntp groups where is a series of USENET newsgroups, separated by blanks or (preferably) commas. This command controls which newsgroups you will receive. By default you will receive all newsgroups. You should probably use this command to restrict the number of groups, unless you have unlimited disk space. The "*" and "!" signs are supported. For example nntp groups rec.ham-radio*,!rec.ham-radio.swap will make you receive all groups whose name matches rec.ham-radio except rec.ham-radio.swap. The NNTP client asks the server for the Message-ID's of any new articles that have arrived after a certain date. The client compares the Message-ID's that it gets from the server with ID's of old articles. These old ID's are kept in the /spool/news/history file. The client then requests those articles that it does not already have. The date and time when a server was last contacted is stored in the file /spool/news/nntp.dat. The date can be changed manually if necessary. Don't forget to create the /spool/news directory, by the way. The articles are not stored as separate files, as is common in UNIX. Instead, they are converted to mailbox format and stored in one separate file for each newsgroup. The reason for this is that you are then able to use the BBS already inherent in NOS to read the articles. It is also possible to use BM for that matter. A typical entry in the /spool/areas file would be: rec/ham-radio/packet The packet radio newsgroup The client still needs some enhancement. In particular, it would be nice if it could post articles too. I might have to use the stat() call. Has it been fixed for DesqView yet? The client should also make it possible to just grab the header of an article, and let the user decide if the rest of the article should be transfered. But this would require a special news reader utility. The source for this is vailable with ftp from sics.se. The filename is archive/packet/ka9q/nos/nntp.arc. /Anders SM0RGV ---------------------------------------------------------------------------- The new changes (as of May 12, 1990) are described next, followed by older changes: Newsgroup name adjustment Previously, newsgroups whose names contained characters that were not valid in DOS filenames would not get created. I now map any character that is nonalphanumeric and not '-' into an underscore ('_'). Locking of nntp.dat history files Previously, it was possible for a session to open while another was still in progress; the result would be that two processes would be trying to update these files. I've added appropriate mlock() and rmlock() calls to prevent this. If nntp.dat is locked when I try to first read it, I close the session and try again later. If it's locked when I try to update it, I simply don't update it; the next session to that server will get the old date in the NEWNEWS command, but the old message ids will be found in the history file and so not transferred. If the history file is locked, I abort the session; this is not a great solution, but is better than the alternative. Updating of nntp.dat changed Instead of overwriting it in place (which for reasons unknown didn't seem to work for me), I now recreate it. Enhanced NNTP DIR command You can now give a second argument, which becomes the Newsdir directory. Changed date stamp on 'From ' lines used to be mm/dd/yy hh:mm:ss, now it's just elapsed seconds since origin of GMT. Make it both easier to parse and easier for expire to work with. Buffers in getarticle() now malloc'd used to be on stack -- Phil pointed out it's better to mallocw() them, so I do. The following describes my changes to Anders Klemets' NNTP client for the KA9Q NOS package. The changes described herein are as of May 6, 1990. --Bernie Roehl broehl@sunee.waterloo.edu NNTP TRACE command The syntax is the same as for SMTP TRACE. There are four levels of detail that can be specified: 0 - no tracing 1 - only serious errors reported 2 - same as 1, plus transient errors reported 3 - same as 2, plus session progress reported 4 - same as 3, plus actual received articles displayed Note that (1) is the default, and corresponds to the way things were in Klemets' original code. More error messages I've added some error reporting (most of it subject to the nntp trace setting). NNTP DIR command Allows you to set a spool directory other than the mail directory. Note that this is *not* the "Newsdir" directory (where history and nntp.dat are kept) but rather the spool directory where the articles themselves are kept. The idea is that you could store articles on another drive. Connection windows You can now specify a range of times at the end of an "nntp add" command, in the form "nntp add localserver 600 22:00-23:00", which would cause the nntp connections to be made only between 10 pm and 11 pm local time. (NOTE: uses gettime() call, which is DOS-specific). Site-specific newsgroups You can now override the "nntp groups" setting for specific servers. For example, nntp groups !rec.pets.gerbils nntp add myfeed 1800 nntp add anotherfeed 7200 2:00-5:00 rec.pets.gerbils would take a full newsfeed from site "myfeed" every half-hour, but would not take rec.pets.gerbils. The group rec.pets.gerbils is brought in from site "anotherfeed" every 2 hours, but only connecting between 2 and 5 am local time. Accumulating NNTP ADD commands Because additional information is now added on to NNTP, the commands can get quite long. Therefore, "nntp add" commands now merge their values. For example, nntp add myfeed 1800 rec.pets.gerbils,6:00-9:00,7:30-9:30 nntp add myfeed 7200 comp.binaries.eniac nntp add myfeed 600 nntp list will produce the following: myfeed (595/600 7:30-9:30) rec.pets.gerbils,comp.binaries.eniac Note that the interval value *replaces* the previous one, as does the window setting; the groups accumulate (up to 512 bytes total). To reset the group list, "nntp drop" the entry and re-add it. File Locking The mlock() and rmlock() functions within NOS are now used to lock the news files while they're being updated. This is for use in conjunction with a (yet-to-be-written) news-expiry program. Date/time stamp on 'From' lines 'From' lines in the news files now have the date and time appended, in the form MM/DD/YY HH:MM:SS. This for use with a (yet-to-be-written) news expiry program. (Note: this is the *arrival* date of the article, not the date it was posted on!) Increased stack space Even in the original nntpcli.c, the stack allocated was not really enough. I've bumped it to 3k (up from 1k) because of all the array[NNTPMAXLEN]'s in getarticle() and elsewhere. Updating of nntp.dat only if the session is succesful Otherwise, we might lose articles. That's about it. Be sure to read Anders Klemets' original description, in klemets.txt, and of course read through the code to see how it works :-) If you have any questions about the code, send mail to me at broehl@sunee.waterloo.edu, or to tcp-group@ucsd.edu. I'm working on an expiry program (as time permits...), and am thinking about how to implement posting (using IHAVE). Will keep you posted.